Stochastic Optimization for Multiview Representation Learning using Partial Least Squares
نویسندگان
چکیده
Partial Least Squares (PLS) is a ubiquitous statistical technique for bilinear factor analysis. It is used in many data analysis, machine learning, and information retrieval applications to model the covariance structure between a pair of data matrices. In this paper, we consider PLS for representation learning in a multiview setting where we have more than one view in data at training time. Furthermore, instead of framing PLS as a problem about a fixed given data set, we argue that PLS should be studied as a stochastic optimization problem, especially in a “big data” setting, with the goal of optimizing a population objective based on sample. This view suggests using Stochastic Approximation (SA) approaches, such as Stochastic Gradient Descent (SGD) and enables a rigorous analysis of their benefits. In this paper, we develop SA approaches to PLS and provide iteration complexity bounds for the proposed algorithms.
منابع مشابه
Dropping Convexity for More Efficient and Scalable Online Multiview Learning
Multiview representation learning is very popular for latent factor analysis. It naturally arises in many data analysis, machine learning, and information retrieval applications to model dependent structures among multiple data sources. For computational convenience, existing approaches usually formulate the multiview representation learning as convex optimization problems, where global optima ...
متن کاملA Uniied Approach to Pca, Pls, Mlr and Cca
This paper presents a novel algorithm for analysis of stochastic processes. The algorithm can be used to nd the required solutions in the cases of principal component analysis (PCA), partial least squares (PLS), canonical correlation analysis (CCA) or multiple linear regression (MLR). The algorithm is iterative and sequential in its structure and uses on-line stochastic approximation to reach a...
متن کاملSimplex design method in simultaneous spectrophotometric determination of silicate and phosphate in boiler water of power plant and sewage sample by partial least squares
Partial least squares modeling as a powerful multivariate statistical tool was applied tothe simultaneous spectrophotometric determination of silicate and phosphate in aqueoussolutions. The concentration range for silicate and phosphate were 0.02-0.6 and 0.4-3 μg ml-1,respectively. The experimental calibration set was composed with 30 sample solutions using amixture design for two component mix...
متن کاملEnsemble manifold regularized sparse low-rank approximation for multiview feature embedding
In computer vision and pattern recognition researches, the studied objects are often characterized by multiple feature representations with high dimensionality, thus it is essential to encode that multiview feature into a unified and discriminative embedding that is optimal for a given task. To address this challenge, this paper proposes an ensemble manifold regularized sparse low-rank approxim...
متن کاملWithout-Replacement Sampling for Stochastic Gradient Methods: Convergence Results and Application to Distributed Optimization
Stochastic gradient methods for machine learning and optimization problems are usually analyzed assuming data points are sampled with replacement. In practice, however, sampling without replacement is very common, easier to implement in many cases, and often performs better. In this paper, we provide competitive convergence guarantees for without-replacement sampling, under various scenarios, f...
متن کامل